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Abstract 



A multi-class single-server system with general service time distributions is studied in a 
moderate deviation heavy traffic regime. In the scaling limit, an optimal control problem 
associated with the model is shown to be governed by a differential game, that can be 
, explicitly solved. While the characterization of the limit by a differential game is akin to 

results at the large deviation scale, the analysis of the problem is closely related to the 
, much studied area of control in heavy traffic at the diffusion scale. 
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1 Introduction 

Models of controlled queueing systems have been studied under various scaling limits. These 
include heavy traffic diffusion approximations, which are based on the central limit theorem 
(see [8], [5] and references therein) and large deviation (LD) asymptotics (see eg., [I], [2] and 
references therein). To the best of our knowledge, the intermediate, moderate deviation (MD) 
scale has not been considered before in relation to controlled queueing systems. In this paper 
we consider the multi-class G/G/l model in a heavy traffic MD regime with a risk-sensitive 
type cost of a general form, characterize its asymptotic behavior in terms of a differential 
game (DG), and solve the game. In a special but important case, we also identify a simple 
policy that is asymptotically optimal (AO). The treatment in the MD regime shares important 
characteristics with both asymptotic regimes alluded to above. It is similar to analogous results 
in the LD regime, in that the limit behavior is indeed governed by a DG. The DG itself is 
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closely related to Brownian control problems (BCP) that arise in diffusion approximations. In 
particular, the solution method by which BCP are transformed into problems involving the 
so-called workload process, turns out to be useful for solving these DG as well. 

Treatments of queueing models in the MD regime without dynamic control aspects include 
the following. In [23j , Puhalskii and Whitt prove LD and MD principles for renewal processes. 
Puhalskii [22] establishes LD and MD principles for queue length and waiting time processes 
for the single server queue and for single class queueing networks in heavy traffic (Puhalskii 
refers to this regime as near heavy traffic, to emphasize that the deviations from critical load 
are at a larger scale than under standard heavy traffic; we will use the term heavy traffic in this 
paper). Majewski [21] treats feedforward multi-class network models with priority. Wischik 
[26j (see also [IB]) illuminates on various links between results on queueing problems in LD 
and MD regimes, as well as similarities between MD and diffusion scale results, particularly 
the validity of results such as the snapshot principle and state space collapse. Based on these 
similarities he conjectures that the well-established dynamic control theory for heavy traffic 
diffusion approximations should have a parallel at the MD scale. Our treatment certainly 
confirms this expectation, at least for the model under investigation. Cruise [9] considers LD 
and MD as a part of a broader parametrization framework for studying queueing systems. 

In the model under consideration (see the next section for a complete description), cus- 
tomers of / different classes arrive at the system following renewal processes and are enqueued 
in buffers, one for each class. A server, that may offer simultaneous service to the various 
classes, divides its effort among the (at most) I customers waiting at the head of the line of 
each buffer. The service time distributions depend on the class. The problem is to control these 
fractions of effort so as to minimize a cost. MD scaling is obtained by considering a sequence 
b n , where b n — > oo, y/n/b n — > oo. The arrival and service time scales are set proportional to 
a large parameter n, with possible correction of order b n y/n. Denoting by X™(t), the number 
of class-i jobs in the n-th system at time t, a scaled version is given by X n = (b n yfn)~ 1 X n . 
Moreover, a critical load condition is assumed, namely that the limiting traffic intensity is one. 
The cost is given by 

— log E{e & ™ [ ^o T h ^ n (t))dt+9(x n (T))] } } 

where T > 0, and h and g are given functions. 

This type of cost is called risk-sensitive (see the book by Whittle |25j). The optimal control 
formulation of a dynamical system with small noise goes back to Fleming [15] , who studies the 
associated Hamilton-Jacobi equations. The connection of risk-sensitive cost to DG was made 
by Jacobson [20J . The study of risk-sensitive control via LD theory and the formulation of the 
corresponding maximum principle are due to Whittle [24] . Various aspects of this approach 
have been studied for controlled stochastic differential equations, for example, [12], [TO], [17] . 
For queueing networks, risk sensitive control in the LD regime has been studied in [TO], [I], [2]. 
Operating a queueing system so as to avoid large queue length or waiting time is important in 
practice, for preventing buffer overflow and assuring quality of service. A risk-sensitive criterion 
penalizes such events heavily, and thus provides a natural way to address these considerations. 
Further motivation for this formulation is that the solution automatically leads to robustness 
properties of the policy (see Dupuis et al. [H]). Note that working in MD scale leads to some 
additional desired robustness properties. Namely, since the rate function in this case typically 
depends only on first and second moments of the underlying primitives, the characteristics of 
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the problem are insensitive to distributional perturbations which preserve these moments. The 
price paid for working in MD scale is that a critical load condition has to be assumed for the 
problem to be meaningful (as it is in diffusion approximations but not in LD analysis). 

The DG governing the limit behavior can be solved explicitly, a fact that not only is useful 
in characterizing the limit in a concrete way, but also turns out to be of crucial importance 
when proving the convergence. To describe the game (see Section 2 for the precise definition), 
consider the dynamics 

tp(t) = x + yt + [ (A(s) - fi{s))ds + rj(t) G 
Jo 

Here x is an initial condition, y is a term capturing the order b n ^/n time scale correction 
alluded to above, and A and fx represent perturbations at scale b n /y/n of arrival and service 
rates, respectively. These are functions mapping [0, T] — > K+, controlled by player 1. Next, 
T] : [0,oo) — > is a function whose formal derivative represents deviations at scale b n /y/n 
of the fraction of effort dedicated by the server to each class. This function is controlled by 
player 2, and is regarded admissible if (a) for all t, ip(t) E (b) 9 • ??(0) > 0, and (c) 9 ■ r] is 
nondecreasing, where # = (^-,...,y^-)is what is often called the workload vector in the heavy 
traffic literature. The cost, which player 1 (resp., 2) attempts to maximize (minimize) is given 

I h(<p(s))ds + g(<p(T)) - [ Y^iaMsf + bi^{s) 2 ]ds, (1.1) 
io Jo 

where aj and 6j are positive constants. 

It is instructive to compare this to the game obtained under LD scaling. The form presented 

here corresponds to the multiclass M/M/l model, following [2] (the setting there includes 

multiple, heterogenous servers, but the presentation here is specialized to the case of a single 

server). One considers 

(p = r[ip], ip(t) = x + / (A(s) - u(s) • ji(s))ds, 

Jo 

where r is the Skorohod map with normal reflection on the boundary of the positive orthant, A 
and p, are functions [0, T] — > [0, oo) 7 , representing perturbations at the LD scale, and controlled 
by a maximizing player; u : [0,T] — > S where S = {s G [0,1]^ : ^ Sj = 1} is controlled by 
minimizing player representing fraction of effort per class, and • denotes the entrywise product 
of two vectors of the same dimension. The cost here takes the form 

T h(<p(s))d8 + g(<p(T)) - [ T [l ■ l(X(s)) + u(s) ■ ms))]ds, (1.2) 

Jo 

where I and I represent LD cost associated with atypical behavior (see [2] for more details). The 
paper [2] provides a characterization of the game's value in terms of a Hamilton-Jacobi-Isaacs 
(HJI) equation. However, it is not known if the game can be solved explicitly. In contrast, 
the game associated with MD turns out to be explicitly solvable, as we show in this paper. 
The reason for this is that while in the LD game the last term of the cost (|1.2p involves both 
(A,/Z) and u, the corresponding term in (jl.ip involves only (A, /2), not rj. Hence this term 
plays no role when one computes the optimal response rj to a given (A, /2) (it does when one 
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optimizes over (A, fl)). This optimal response is computed via projecting the dynamics in the 
direction of the workload vector, and using minimality considerations of the one-dimensional 
Skorohod problem. In fact, the optimal response rj to (A, p) is precisely the one that arises in 
the diffusion scale analysis of the model, used there to map the Brownian motion term to the 
optimal control for the BCP. Thus the link to diffusion approximations is strong. 

In [2] (following the technique of p]), the convergence is proved by establishing upper and 
lower bounds on the limiting risk-sensitive control problem's value in terms of the lower and, 
respectively, upper values of the DG. The existence of a limit is then argued via uniqueness of 
solutions to the HJI equation satisfied by both values. The arrival and service are assumed to 
follow Poisson processes and the convergence proof uses the form of the Markovian generator 
and martingale inequalities related to it. Since in the MD regime the performance depends 
only on the first two moments of the primitives, these moments carry all relevant information 
regarding the limit (under tail assumptions), and so in this paper we aim at general arrival and 
service processes. As a result, the tools based on the Markovian formulation mentioned above 
can not be used. The approach we take uses completely different considerations. The asymp- 
totic behavior of the risk-sensitive control problem is estimated, above and below, directly by 
the DG lower value (the corresponding upper value is not dealt with at all in this paper). This 
is made possible thanks to the explicit solvability of the game. More precisely, the arguments 
by which the game's optimal strategy is found, including the workload formulation and the 
minimality property associated with the Skorohod map, give rise, when applied to the control 
problem, to the lower bound. The proof of the upper bound is by construction of a particular 
control which again uses the solution of the game and its properties. Note that this approach 
eliminates the need for any PDE analysis. 

The control that is constructed in the proof of the upper bound is too complicated for 
practical implementation. However, in the case where h and g are linear (see Section [5] for 
the precise linearity condition), a simple solution to the DG is available, in the form of a fixed 
priority policy according to the well-known c/i rule. As our final result shows, applying a 
priority policy in the queueing model, according to the same order of customer classes, is AO 
in this case. 

To summarize the main contribution of the paper, we have 

• Provided the first treatment of a queueing control problem at the MD scale, 

• Identified and solved the DG governing the scaling limit for quite a general setting, 

• Proved AO of a simple policy in the linear case. 

The following conclusions stem from this work 

• Techniques such as the equivalent workload formulation, which have proven powerful for 
control problems at the diffusion scale, are useful at the MD scale. They are likely to be 
applicable in far greater generality than the present setting. 

• Although control problems at MD and LD scales are both motivated by similar rationale, 
MD is evidently more tractable for the model under consideration, and potentially this 
is true in greater generality. 

We will use the following notation. For a positive integer k and a, b E M. k , a • b denotes 
the usual scalar product, while || • || denotes Euclidean norm. We denote [0,oo) by R+. For 
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T > and a function / : [0,T] -> P fc , let = sup se[0jt] ||/(s)||, t E [0,T]. When k = 1, 

we write |/|* for ||/||£. We sometimes write ||/||* for ||/||y when there is no ambiguity about 
T. Denote by C([0,T],R k ) and V([0, T], R k ) the spaces of continuous functions [0,T] -)• E fc 
and, respectively, functions that are right-continuous with finite left limits (RCLL). Endow the 
space V([0,T],M. k ) with the J\ metric, defined as 



d(<p, <p') = inf (||/||° V sup \\<p(t) - ip'(f(t))\\) , <P, <f' G ^([0, (1.3) 
where T is the set of strictly increasing, continuous functions from [0, T] onto itself, and 

/w - /(«: 



sup 

0<s<t<T 



log 



t - S 



(1.4) 



As is well known [6], P([0, T],R fe ) is a Polish space under this metric. 

The organization of the paper is as follows. The next section introduces the model and an 
associated differential game and states the main result. In Section 3 we find a solution to the 
game and describe properties of it that are useful in the sequel. Section 4 gives the proof of 
the main theorem. In Section 5 we discuss the case of linear cost and identify an AO policy. 
Finally, the appendix gives the proof of a proposition stated in Section 2. 



2 Model and results 
2.1 The model 

The model consists of I customer classes and a single server. A buffer with infinite room is 
dedicated to each customer class, and upon arrival, customers are queued in the corresponding 
buffers. Within each class, customers are served at the order of arrival. The server may only 
serve the customer at the head of each line. Moreover, processor sharing is allowed, and so the 
server is capable of serving up to I customers (of distinct classes) simultaneously. 

The model is defined on a probability space (f2, J 7 , P). Expectation with respect to P is 
denoted by E. The parameters and processes we introduce will depend on an index n E N, that 
will serve as a scaling parameter. Arrivals occur according to independent renewal processes, 
and service times are independent and identically distributed across each class. Let X = 
{1,2, . . . ,/}. Let A™ > 0, n € N, i E X be given parameters, representing the reciprocal mean 
inter-arrival times of class-i customers. Given are I independent sequence {IAi(l) : I £ N}j S x, 
of positive i.i.d. random variables with mean E[L4j(l)] = 1 and variance cf IA = Var(7^4j(l)) E 
(0,oo). With J2\ = 0) the number of arrivals of class-i customers up to time t, for the n-th 
system, is given by 

A?(t) = Bup{l>0 :£^P<*}, t>0. 
k=i 1 

Similarly we consider another set of parameters /i™ > 0, n E N, i E X, representing reciprocal 
mean service times. We are also given I independent sequence {STi(l) : I E N}j g x of positive 
i.i.d. random variables (independent also of the sequences {L4j}) with mean E[<STj(l)] = 1 and 
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variance af ST = Var(5Tj(l)) G (0, oo). The time required to complete the service of the Z-th 
class-i customer is given by ST j fj,™ , and the potential service time processes are defined as 

Sr(t) = sup{/>0 :£^P<*}> «>0- 
fc=i 

We consider the moderate deviations rate parameters {b n }, that form a sequence, fixed through- 
out, with the property that limfo n = oo while lim = 0, as n — > oo. The arrival and service 
parameters are assumed to satisfy the following conditions. As n — > oo, 

• -> Ai G (0,oo) and ^ — ^ Mi G (0,oo), 
' A ? := " ™ A *) ^ G (-00,(30), 

• := s^O^ - -> ^ G (-°°« °°)- 

Also the system is assumed to be critically loaded in the sense that ^{ Pi = 1 where Pi = jf: 
for i 6l. 

For i G X, let X" be a process representing the number of class-i customers in the n-th 
system. With S = {x = (xi, . . . , xj) G [0, 1] J : £ «i < l }, let 5™ be a process taking values in 
§, whose i-th component represents the fraction of effort devoted by the server to the class-i 
customer at the head of the line. Then the number of service completions of class-i jobs during 
the time interval [0, t] is given by 

D?(t) := S?CZ?(t)), (2.1) 

where 

2?(t)= f B?{s)ds (2.2) 
J o 

is the time devoted to class-i customers by time t. The following equation follows from foregoing 
verbal description 

X?(t) = X? (0) + A?(i) - SrCZf(i)). (2.3) 

For simplicity, the initial conditions A""(0) are assumed to be deterministic. Note that, by 
construction, the arrival and potential service processes have RCLL paths, and accordingly, so 
do D n and X n . 

The process B n is regarded as a control, that is determined based on observations from 
the past (and present) events in the system. A precise definition is as follows. Fix T > 
throughout. Given n, the process B n is said to be an admissible control if its sample paths lie 
in P([0,T],R J ), and 

• It is adapted to the filtration 

o-{A?{s),S?{T?{s)),i£Z, S <t}, 

where T n is given by f)2.2[> ; 
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• For i E X and t > 0, one has 

= implies Bf(t) = 0, (2.4) 

where X n is given by (|2.3|) . 

Denote the class of all admissible controls B n by 05 n . Note that this class depends on A n and 
S n , but we consider these processes to be fixed. It is clear that this class is nonempty, as one 
may obtain an admissible control, for example, by setting B n = identically. 

We next introduce centered and scaled versions of the processes. For i S X, let 

AUt) = -r^=(Am ~ m S?(t) = -r± 7 =(S?(t)-tft), X?(t) = -r^XfV). (2.5) 
o n y n On\ n o n ^/n 



It is easy to check from (|2.3[) that 

Xf (t) = X?(0) + yft + A^(t) - S?(T?(t)) + Z?(t), (2-6) 

where we denote 

zm = El^ {pit _ T n {t)l y n = yn_ p .~n (2 ?) 

n b n 

Note that these processes have the property 

—^Zf starts from zero and is nondecreasing, (2-8) 

i ^ 

thanks to the fact that ^ Bf < 1 while Yli Pi = 1- Clearly X" is nonnegative, i.e., 

X?(t) > t > 0, i e X. (2.9) 

We impose the following condition on the initial values: 

X n (0) -> x G Rj. as n -> oo. 

The scaled processes ( J 4 n ,S' n ) are assumed to satisfy a moderate deviation principle. To 
express this assumption, let Ifc,/c = 1,2, be functions on T>([0,T],M I ) defined as follows. For 
t/; = (Vi,...,^)eP([o,T],R J ), 

\ X^=i — fn ^1 ( s )^ s ^ an V'i are absolutely continuous and -0(0) = 0, 

Il(^) = < Ai °"i,/A 

I oo otherwise, 



and 



~ y^f=i — i — fn ipi{s)ds if all ^ are absolutely continuous and ^(0) = 0, 



2 ^t-i ^" i>ST 

oo otherwise 



Letl(^) = Ii(V' 1 )+Il2(^ 2 ) for ip = i^ 1 ,^ 2 ) £ £>([0, T], M 2/ ). Note that I is lower semicontinuous 
with compact level sets, properties used in the sequel. 
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Assumption 2.1 (Moderate deviation principle) The sequence 

(A n ,S n ) = (A?,...,AJ,S?,...,Sy), 
satisfies the LDP with rate parameters b n and rate function I in T>([0, T], M. 21 ); i.e., 

• For any closed set F C V{[0,T],R 21 ) 

limsup-^logP((I n ,5 n ) G F) < - inf IN,), 

0% ip£F 

• For any open set G C V([0,T],R 21 ) 

liminf ^logP((i n ,,S n ) 6G)> - inf IN). 
o„ V>eG 

Remark 2.1 It is shown in I23f that each one of the following statements is sufficient for 
Assumption \2.1\ to hold: 

• There exist constants u > 0, /3 G (0,1] such that E[e Uo( - IA ^],E[e Uo( - ST ^} < oo, i £ I, 
and 6n 2 n^/ 2 — > oo; 

• For some e > 0, E[(L4j) 2+e ], E[(£T;) 2+e ] < oo, i € Z, and fc~ 2 logn -»• oo. 

To present the risk-sensitive control problem, let h and g be nonnegative, continuous func- 
tions from to R, monotone nondecreasing with respect to the partial order a < 6 iff 
6 — a G Rj_. Assume that /i, g have at most linear growth, i.e., there exist constants c±, C2 such 
that 

+ h(x) < ci + c 2 . 

Given n, the cost associated with the initial condition X n (0) and control B n G < B n is given by 

' e bi\SZ h{X n (s))ds+g(X n (T))] \ . (2.10) 

The value function of interest is given by 

V n (X n (0)) = inf J n (X n (0),B n ). 

2.2 A differential game 

We next develop a differential game for the limit behavior of the above control problem. Let 
9 = (4-, . . . , X) and y = (yi, ...,?//) where = \ - pife. Denote V = C ([0, T], R 21 ), the 

subset of C([0,T],R 2/ ) of functions starting from zero, and 

£ = {(e C([0,T],R 7 ) : 9- C starts from zero and is nondecreasing}. 

Endow both spaces with the uniform topology. Let p be the mapping from £>([0,T],R J ) into 
itself defined by 

p[ip] i (t)=il> i (p i t), te[0,T],iel. 
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J n (X n (0),B n ) = TJ logE 



Given ip = (ip l ,ip 2 ) G V and Q G £, the dynamics associated with initial condition x and data 
ip, C is given by 

(Pi(t) = x i + y i t + iP 1 i {t)- p[^ 2 ]i(t)+Ci{t), iel. (2.11) 

Note the analogy between the above equation and equation (|2.6p . and between the condition 
9 ■ £ nondecreasing and property (|2.8[) . The following condition, analogous to property (|2.9[) . 
will also be used, namely 

<Pi(t) > 0, t > 0, i G X. (2.12) 

The game is defined in the sense of Elliott and Kalton [13] . for which we need the notion of 
strategies. A measurable mapping a : V — > £ is called a strategy for the minimizing player if it 
satisfies a causality property. Namely, for every ip = (ip 1 , ip 2 ) , ip = (ip l ,ip 2 ) G V and t G [0,T], 

(ip 1 , p[ip 2 ])(s) = (ip 1 , p[tjj 2 ])(s) for all s G [0,t] implies a[tp](s) = a#](s) for all s G [0,t]. 

(2.13) 

Given an initial condition x, a strategy a is said to be admissible if, whenever ^ G V and 
C = a^], the corresponding dynamics ()2.1ip satisfies the nonnegativity constraint (|2.12p . 
The set of all admissible strategies for the minimizing player is denoted by A (or, when the 
dependence on the initial condition is important, A x ). Given x and (ip,C) G x we define 
the cost by 

c#, ()=[ h(<p(t))dt + g(<p(T)) - Ity), 
Jo 

where (p is the corresponding dynamics. The value of the game is defined by 

V(x) = inf sup c(ip, a[ip]). 

2.3 Main result 

For w G M+, denote 

h*(w) = inf {h(x) : x ■ x = w}, g*(w) = mf{g(x) : x G • x = w}. (2.14) 

We need the following assumption. It is similar to the one imposed in [3], [3], where an 
analogous many-server model is treated in a diffusion regime. 

Assumption 2.2 (Existence of a continuous minimizing curve) There exists a contin- 
uous map f : M+ — > Mj_ such that for all w G M+, 

6-f(w) = w, h*(w) = h(f(w)), g*(w)=g(f(w)). 

As far as solving the game is concerned, this assumption is not required at all (see Remark 
13. 1 H . It is important in the proof of asymptotic optimality. The fact that the same function / 
serves as a minimizer for both h and g may seem to be too strong. We comment in Remark 
14.11 on what is involved in relaxing this assumption. 

Example 2.1 a. The linear case: h(x) = ^ciXi and g(x) = ^diXi, for some nonnegative 
constants ci, d{. If we require that cj^ii = mhijCj/ij and dim = minj<ij//j then the assumption 
holds with f(w) = (0, . . . ,0,wni). This is the case considered in Section\^ 
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b. If h is non- decreasing, homogeneous of degree a,0 < a < 1, and x* E argmin{h(x) : ■ x = 
1}, it is easy to check that f(w) = wx* satisfies the above assumption provided g = dh for 
some non-negative constant d. 

Assumption 2.3 (Exponential moments) For any constant K, 



A sufficient condition for the above is as follows (see the appendix for a proof). 

Proposition 2.1 If there exists uq > such that ~E[e u ° IAl ] and E[e' il ° STi ], i € X, are finite 
then Assumption \2.3\ holds. 

Note that taking f3 = 1 in Remark 12.11 shows that the hypothesis of Proposition 12. II is sufficient 
for Assumption 12.11 as well. 

Our main result is the following: 

Theorem 2.1 Let Assumptions \2.1\ and \2.2\ hold. If either g or h is unbounded, let also 
Assumption\2J\hold. Then lim^^ V n (X n {0)) = V{x). 

Remark 2.2 There is a simpler, equivalent formulation of the game, which avoids the use of 
the time scaling operator p (both formulations will be used in the proofs). Define a functional 
I(V>) = ii^ 1 ) + h{^ 2 ) on V([0,T],R 21 ), where I k ,k = 1,2, are functional on £>([0, T], M 7 ) 
given by Ii = Ii, and, for tp = (ipi, . . . , tpj) G f([0, T],R 7 ), 





i=l 



1 



3 — 

i,ST 



Jq ipf(s)ds if all ipi are absolutely continuous and ?/>(0) = 0, 
otherwise. 



The dynamics of the game <p are now 



<Pi(t) =Xi = ytt + ij\t) - V 2 (t) + Ci(t) > 0. 



A strategy a should now satisfy the following version of the causality property: 



ip(s) = tp(s) for all s £ [0,t] implies a[ip](s) = a[ip](s) for all s € [0,t]. 



Denote the set of all such strategies by A x . Given x and (ip,C) £V x £, let 




where (p is as above. Then the value of the game can also be defined as 



V(x) = inf sup c(ijj, aty]). 
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3 Solution of the game 

In this section we find a minimizing strategy for V, under Assumption 12. 2\ following an 
idea from [19]. Throughout this section, the initial condition x is fixed. Consider the one- 
dimensional Skorohod map r from 2?([0,T],R) to itself given by 

r[z](t) = z(t) - M >(«)A0], t€[0,T}. (3.1) 

se[o,t] 

Clearly, r[z](t) > for all t. Let also 

f \z](t) = - inf \z(s) AO], * 6 [0,7*1. 

s6[0,t] 

It is clear from the definition that, for z,w G P([0, T],R) 

sup l-Tfz] — -T[w]| < 2 sup \z — w\. (3-2) 

[0,T] [0,T] 



The construction below is based on the mapping 7 1 and the function / from Assumption | 
Recall from (|2.1ip that for tp = (tp 1 ,^ 2 ) G V and £ G £, the dynamics of the differential game 
is given by 

v? = £ + C, 

where 

m=x + yt + ^(t)-p[i; 2 }(t), t€ [0,7]. 
We associate with each ip G "P a 4-tuple (<£>,£, C w ) = (vtV 7 ]) £ [V 1 ]) CM) w^]) given by 

£M(t) = x + yt + ^ 1 (t)-p& 2 ](t), t£[0,T], (3.3) 

w[^]=r[9-^]}, (3.4) 

¥#] = /(wM) (3.5) 

CM =¥#]-£[# (3.6) 

Sometimes we also use the notation 

£M(t) =x + yi + V 1 (t)"V' 2 (t), iG[0,T], (3.7) 
*M=r[0-£M], (3.8) 
£#] = /(w[V]) (3.9) 

CM = ¥#]-£[# (3-10) 

Note that CM,?A 2 ] = Ct^.pOT- 

Proposition 3.1 Let Assumption \2.S\ hold. Then £ is an admissible strategy. Moreover, it is 
a minimizing strategy, namely 

F(x) = supc(V,CM)- (3-11) 
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Proof: Let us show that £ is an admissible strategy. Let ip £ V be given and denote 
(V, £, C, = C, w ) W\ ■ Then y? = £ + C, and multiplying ([321) by 0, 

Since 9 • £(0) = • x > 0, it follows that 9 ■ C(0) = 0. Moreover, by definition of P, 9 ■ ( 
is nondecreasing. This shows C £ £. The causality property (|2.13p follows directly from an 
analogous property of P. Next, w{t) > for all t, and, by definition, / maps R+ to R+, whence 
ip(t) £ R+ for all t. This shows that £ is an admissible strategy. 

Now we check that £ is indeed a minimizing strategy. This is based on the minimality 
property of the Skorohod map (see e.g. [71 Section 2]). Namely, if z, r £ T>([0, T] : R), r is 
nonnegative and nondecreasing, and z(t) + r(t) > for all t, then 

z(t) + r(t)>r[z](t), te[o,T}. 

Let a £ A be any admissible strategy. Given tp, let £, £, io) be as before. The dynamics 
corresponding to ^ and £ := a[^] is given by (p = £ + £• Since a is an admissible strategy, we 
have that 

9-<p = 9-£ + 9-(>0, 
and • £ is nonnegative and nondecreasing. Thus by the above minimality property, 

o-<p(t)>r[e-t](t) = w(t), te[o,T}. 

By monotonicity of h, (|3.5|) and Assumption I2.2I 



h(if(t)) > inf{%) : 9-q = 9- <p(t)} 

> inf{%) : 9-q = w(t)} = h(f(w(t))) = h(<p(t)). (3.12) 

A similar estimate holds for g, namely 

g(<p(T)) > g(<p(T)). (3.13) 

As a result, 

sup c(tjj, a[ip]) > sup c(ip, CbP])- 
This proves that £ is a minimizing strategy, namely (|3. 1 1|) holds. □ 

Remark 3.1 a. The game can be solved without Assumption \2.SX Owing to the continuity of 
h and g and using a measurable selection result such as Corollary 10.3 in the appendix of 
there exist measurable functions fh and f g mapping R + to such that for all w £ R+, 

■ fh(w) = 9 ■ f g (w) = w, h*(w)=h(f h (w)), g*H=g(f g (w)), (3.14) 

where we recall the definition (12. 14ft of h* and g* . To construct a minimizing strategy, let £ 
and w be as in (|3.3|) - (|3.4p . Instead of (|3.5p . consider 
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Then define C a $ in* f|3.6j) accordingly (£ and V will also change accordingly). The proof of 
Proposition \3.1\ goes through with almost no change. Indeed, the continuity of f is not used 
in this proof, and inequalities (|3.12p and f)3. 13|) can be obtained by working with fh and f g , 
respectively, instead of f. 

b. Although the continuity that is a part of in Assumption \2.2\ is irrelevant for the game, it will 
be used in the convergence argument leading to the asymptotic optimality result (Theorem \4.2\ ). 
One may, however, consider a relaxation of Assumption [2TM as follows: There exist continuous 
functions fh and f g satisfying (|3.14j) above. Under this relaxed assumption, given a continuous 
path ip G V , the corresponding dynamics <p = with ip as in (|3.15p . may then have a jump 

at time T. The jump makes it more complicated to obtain convergence in Theorem We 
discuss this issue in Remark \4-l\ 

Extension and some properties of £. As a strategy, £ is defined on V . We extend £ and 
C to 

V = V([0,T},R 21 ), 

using the same definitions (|3.6|) and (I3,10|) , Some useful properties related to this map are 
stated in the following result. Given a map m : [0,T] — > some k G N, and r\ > 0, define 
the n-oscillation of m as 

osc^m) = sup{||m(s) — m(t) \\ : \s — t\<T), s,t G [0,T]}. 

For k > 0, define (with || • ||* = || • ||^) 

V(k) = {ip = (ip 1 ,^ 2 ) G V ■ U 1 ]]* + U 2 \\* < k and £[ip](0) G R T + }. (3.16) 



Proposition 3.2 Let Assumption \2.2\ hold. 

i. Given £, ( G T>([0, T], IR 7 ), cp(t) = + £(i) G M. + , 9 ■ ( nonnegative and nondecreasing, one 
has 

j(<p(t)) >j(f(r[0- £] (t))), forj = h,g. (3.17) 

ii. There exist constants 70 and 71 such that for ip G V , 

iicM(t)ii<7o(ii^ii: + ii^iin+7i. (3.18) 

Hi. For if),ip G T>(k), given e > there exists S\ > such that 

IICM -<t$]\\* <£ provided W^ 1 - ip 2 \\* + ||^ 2 - i> 2 \\* < $1. (3.19) 

iu. For any ^ G "D(k), given e > i/iere exisi <5 > and rj > such that 

0SC -q{CW\) < e provided osc v (ip) < 5. (3.20) 

Proof: i. The argument leading to (I3.12|) and (13.130 is seen to be applicable for this extended 
map, giving (|3.17p . 

ii. Denote 9* = minj g x#j and 9* = maxj g j0j. Then Assumption 12.21 implies that < 
y-w for w > 0. Let 70 = y/l(^p — h 1) and 71 = 70 Si=i( x « + T\yi\). Then for t G [0, T], using 
pTfjl - prn]) . <|3TT5|> holds. 
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iii. Using (|3.9p and (|3. 18j) . for every k there exists a constant /3 = /3(k) such that, for all 

fcmw* < 

Thus given e > we can find 5 = 6(n,e) such that — f(w2)\\ < | if \wi — u^l < 

5 and Wi £ [0, /3(n)]. Also using the relation = r[6 • £[?/>]], we have for ip,ip £p 

iw[^] - w[$r < cunv 1 - v^ 2 ir + nv> 2 - ^ 2 in, 

for some constant c\. Choosing 8\ = 5i(k,e) sufficiently small, for ip,ip £ "D(k) we have, with 
tp and (p denoting the dynamics corresponding to {if),C[if}]) and resp., (ip, C[ip]), 

iiy,-^ii*<| if H^-^r + ii^-^ii*^. 

Using the above estimate and f|3. 10|) we have (|3.19p . 

iv. Property (|3.20p follows directly from the definition of r, definitions (|3,7p - (|3,10p . and 
the continuity of /. □ 

4 Proof of Theorem 12.11 
4.1 Lower bound 

Theorem 4.1 Let Assumptions\2.1\ and\2.2\ hold. Then liminf V n (X n (0)) > V(x). 

In the proof, we choose any path ip £ V and show that for any nearly optimal policy, the 
paths X n {-) can be controlled suitably for (A n , S n ) close to ip. We find a constant G > such 
that for 9 ■ Z n > G the lower bound becomes trivial by using the monotonicity of h and g, and 
for 9 ■ Z n < G, the optimality of £ gives the required estimates. 

Proof: Fix tp = V> 2 ) G V. Let d(; •) be as in flOJ). Define, for r > 0, 

Ar = e V{[0,T},R 21 ) : d(ip,$)<r}. 

Since ip is continuous, for any v\ G (0, 1) there exists r > such that 

V> G A implies - ^||* < n. (4.1) 

Define 9 n = ■&). Then 9 n — > 9 as n — > oo. Now, given < e < 1, choose a 

sequence of policies {B n } such that 

U n (X n (0)) + e > J n {X n (0), B n ) and B n G <B n for all n. (4.2) 

Recall that 

J"(-r*(0),B n ) = llogE[ e 6 «[/o T K^W)^(^(T))] ]5 (43) 
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where 

X?(t) = X?(0) + y?t + Af(t) - S?(T t n (t)) + Zm, (4-4) 

zm = t^^ {p . t _ T n m T n {t) = r B n (s)dg . (45) 
n On Jo 

For G > 0, define a random variable r n by 

r I 

r n = inf{t > : 9 n ■ Z n (t) > G} A T = inf \t > : > G j A T. 

By (|2.8p . # n • Z n is nondecreasing and continuous and hence 

9 n ■ Z n (t) < G for t < r n 
6 n -Z n (t) > G for t>r n . 

Consider the event (A n , S n ) G A r . Under this event, for t > r n , 

P .X n (t)>-\\0 n \\( Ko + 2\$\\*) + G, 

where kq is a constant (not depending on n or G) and we used (|4.ip and the boundedness of 
X n (0) and A" — pijlf- Since also 6 n converges, we can choose a constant k\ such that, on the 
indicated event, 

6 n -X n {t) > -ki + G, t>r n . (4.6) 

Next, let w = w[ip], ip = <p [ip], ( = C[ip] (see (|3.3p - (|3.6p ). Note that ip is the dynamics 
corresponding to (tp,C), namely 

<Pi(t) = x t + Vi t + #(t) - $( Pi t) + Ci(t). (4.7) 

For any k > define a compact set Q(k) as 

Q(«) = { g eK 7 + : 2q- 6 < k}. 

Choose k large enough so that 

h{z) > \h(ip(-))\*T and g{z) > g(<p(T)) 

for all z G Q c (k). To see that this is possible note that h(f(-)) is nondecreasing, and for 
2 G Q c (k) 

> miii{%) : 6 ■ q = 6 ■ z} = h(f(9 ■ z)), 
where we use the definition of /. Thus 

h(z) > h(f( K /2)), 

where we use the monotonicity of h(f(-)). Since tp(t), t G [0, T], is bounded, so is w(t), 
t G [0, T], by continuity of r. Choosing k = 2\w\^ and using again the monotonicity of h(f(-)), 
gives the claimed inequality for h. A similar argument applies for g. 
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Since 9* := mirij 9i > 0, we can choose no large enough to ensure that 9f < 29i for alii G X 
and n > no- Now if we choose G in (|4.6p large enough so that — k\ + G > k, we have for 
t > T n ,n> n , 

29-X n (t) > 9 n -X n (t) > k, 
and hence by our choice of k we have on the indicated event, for all t > r n , 

h(X n (t)) > \h(cp)\* and g(X n (t)) > g(tp(T)) for all sufficiently large n. (4.8) 

Now we fix G as above and consider t < r n , on the same event (A n ,S n ) G A r . The 
nonnegativity of Xf and (14.4p imply a lower bound on each of the terms Z™, namely 

Zf(i) > -X"(0) - yft - i?(t) + SP(2?(i)). 

Therefore using (|4.ip there exists a constant K2 such that for all sufficiently large n, Zf(t) > 
— k>2- Combining this with the definition of r n in terms of G, we have for t < r n and all large 

\\Z n (t)\\ <k 3 . (4.9) 
Consider the stochastic processes ^ n ,^> n , Z n , with values in Mr, 

8?(i) = Af(tAr n ), 

§?(t) = x t -X?{Q) + { yi -y?)t+~S?{T?{tAT n ))-(l-^9?)Z?{tAT r ^ 

Z?(t) = »wz?{t). 

Then by (jOj) . 

X?(t) =Xi + Vi t + V?(t) - + Zf (*), t G [0, r„]. (4.10) 

Note that q? n ,St? n have RCLL sample paths, and consider <£> n = (p[^ n ,^ n }. Then 

<p n (t) = 2 ; + 2/t + <P' n (t)-^ n (t) + C[^ n ,^ n ](i)- (4-11) 

Let us now apply Proposition E^i) with = x + yt + fl^(t) - ^ n (t) and ( = Z n . Note that 
X n = £ + C takes values in R^_, by definition, and that 9-Z n is nonnegative and nondecreasing, 
by ([22]). Moreover, by definition of Cp (see (|577|l - (|53|l ). <2> n = f{r[9 • £]). Hence ([3TT7D gives 

> /i(* n (i)) and 5 (X n (t)) > /i(^ n (t)), 4 G [0,r n ]. (4.12) 

Let k 4 = H-0II*. By (jlTTj) . on the indicated event, (i n ,S n ) G P(2(l + k 4 )) where we 
recall definition (|515|) . Note that x + ^ n (0) - ^ n (0) = i" n (0) G and, from that 
(<P n ,& n ) G 2?(2(2 + k 4 )) for all large n. Since < Bf (a) < 1, 7; n (s) G [0,t„] for all s G [0,r n ]. 
Hence from (14.ip we have for (Ji n , S" 1 ) G A r 

sup _ S?(T?(t))\ < n + sup |$( Pi t) - 

[0,r n ] [0,r n ] 

Again using the continuity of V' 2 , we can choose r2 > small enough such that osc^h/' 2 ] < r\. 
Since ^= — > 0, we note from (|4.9p that for all large n, and all i, sup[ 0)Tn ] \pit — T™(t)\ < r%. 

Since X n (0) -> x, y n -> y and n -> 0, it follows that 
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for all large n. Now taking k = 2(2 + K4), we choose r\ sufficiently small (see (|3.19p ) so that 
for all n large we have 

\\C$}-CW n ,# n ]\\* Tn <e- 

Now choosing r < e/(3\/7) and using (|4.7|) and (|4.11|) . for (A n ,S n ) G A r and all large n, we 
have 

\\<p-$ n \\* n <4£. (4.13) 

Let K5 = + 4). Denote by [resp., ui g ] the modulus of continuity of h [resp., g] over 

{q : < K5}. Then by (I4.12j) . on the indicated event, for all large n, 



h(X n (s))ds > / h($ 







n {s))ds> I ' " h{y(s))ds-Tuj h (Ae). 
Jo 



Combined with f|4. 8|) this gives 



h(X n (s))ds> h(v(s))ds-Tio h (Ae). 
Jo 

A similar argument gives 

g(X n (T)) = g(ip(T))x { T< Tn } + g(v(T))x { T>r n} > g(<p(T)) - u g (4s). 
Hence for all large n, 



> E 



bl{fTh(X"(s))ds+g(X"(T))] 



X{(A n ,S n )eA r } 



MIo H<p(s))ds+g(<p(T))-a(e)] , 



where a(e) = [Tcj/ l (4e) +w 9 (4e)] — > as e — > 0. We now use Assumption 12.11 Since A r is open, 

¥((A n ,S n ) G A-) > e" 6 ^™^ 6 ^ I ^) +£ l > e - fe nP I W i )+ £ ] 
holds for all sufficiently large n. Hence we have from (|4.2p and (|4.3p that for all large n, 
y n (X n (0))+e > J(X n (0),B n ) 



> 



T 



h(<p(s))ds + g(ip(T)) - 1$) - a(e) - e. 



Therefore 



liminf y n (X n (0)) > c(^,C[^]) - a(e) - 2e, 



and letting e — > 0, we obtain liminfn^oo y n (X n (0)) > c(-0, C["0D- Finally, since tp G T 7 is 
arbitrary we have from (I3TTT) that liminf^oo F n (X n (0)) > V(x). □ 
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4.2 Upper bound 

Theorem 4.2 Let Assumptions \2.1\ and \2.2\ hold. If either g or h is unbounded, let also 
AssumytionWMhold. Then limsup V n (X n (0)) < V(x). 

The proof is based on the construction and analysis of a particular policy, described below 
in equations (|4.18p - (14,23p . To see the main idea behind the structure of the policy, refer to 
equations (|2.6p and (|2-7p . which describe the dependence of the scaled process X n on the 
stochastic primitives A n , S n , and the control process B n (recall from (I2.2p that T n is an 
integral form of B n ). Because of the amplifying factor \fnjb n which appears in the expression 
(I2T7D in front of 

Pi t-mt)= [\ Pi -B?(s))ds, 
Jo 

it is seen that fluctuations of B n about its center p, at scale as small as & n /\Ai, cause order- 
one displacements in X n . Initially, the policy drives the process X n from the initial position 
X n (0) ~ x to the corresponding point on the minimizing curve, f(6-x), in a short time. This 
is reflected in the choice of the constant I applied during the first time interval [0, v ) (see first 
line of (|4.22p ). Afterwards, the policy mimics the behavior of the optimal strategy for the 
game, namely £. This is performed by applying F n (see third line of (j4.22|) ). which consists 
of the response of C, in differential form, to the stochastic data P n (see (|4.19p ). 
Proof: Given a constant A, define 

P 4 = {^D([0,T],1 2/ ) : I(rjj)<A}. (4.14) 

By the definition of I (from Section [2]) , T>a is a compact set containing absolutely continuous 
paths starting from zero (particularly, T>a C V), with derivative having L 2 -norm uniformly 
bounded. Consequently, for a constant M = Ma and all ip £ T>a, one has H^ 1 1|* + HV^II* < M. 
Consider the set V(M + 1) (see (I3.16|) ). let e G (0, 1) be given, and choose 6i,5,rj > as in 
(I3.19P and (|3.20p . corresponding to e and k = M + 1. Assume, without loss of generality, that 
5\ V 5 < e. It follows from the L 2 bound alluded to above, that for each fixed A, the members 
of T>a are equicontinuous. Hence one can choose vq S (0, 77) (depending on A), such that 

osc„ (y>-) < -^L, for all V = 0/>\V> 2 ) eV A , 1 = 1,2, ie 1. (4.15) 
4v 21 

Recall from (jl.3p - (jl.4p the notation d, T and || • ||°. As in the proof of Theorem 14. H we set for 

AS) = {ipeV([0,T],R 21 ) : d(i;,4,)<r}. 
Noting that, for any / € T, 

wm-mw < \\m-^f(t))\\ + u(f(t))-m\\, 

I/O- -It < r( e w°-i), 

it follows by equicontinuity that it is possible to choose V\ > such that, for any xj) E Da-, 

i/> € Amify implies - $\\* < . (4.16) 
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Let V2 = minjfo, v±, |}. Since T>& is compact and I is lower semicontinuous, one can find 
a finite number of members ijj 1 , ijj 2 , . . . , tp N of T>a, and positive constants v^,...,v with 
v k < V2, satisfying T>a C Ufc.A fe , and 



inf{I(^) : ^ e > I(ip k ) 



k = l,2,...,N, 



(4.17) 



where, throughout, A k := ^4„fc(V ;fc )- 

We next define a policy for which we shall prove that the lower bound is asymptotically 
attained. Fix n £ N. Recall <^J^, §F2§ and ([23]) by which 



D n = S n oT ^ 

T? = [ B?{s)ds, 
Jo 

Xf = X?(0)+A?-D? 



(4.18) 



Recall the scaled processes (|2.5p and let also 



D n = S n oT n^ 

P n = {A n ,D n ). 



(4.19) 



The analogy between the queueing system dynamics (12. 6p and the game dynamics (|2.1ip sug- 
gests that the policy should be designed in such a way that f^XS j Q [p i —B^(s))ds Ci[-P n ] holds 
for each i. Equivalently, one should have JjJ Bf{s)ds ~ p^i — ^ fe ^,- ^jP n ](t). A straightforward 
discretization approach fails to provide an admissible control. A version of this approximate 
equality that does define an admissible control is as follows. Denote 



&(a, b) = aX[o,x] («)X[o,i] ( & )' a,beR. 
Let £ = f(x ■ 0) — x and v = ^ A ?. For i £l, assume that .Bf is given by 

B?(t)=C?(t) X{X ?(t)>o h t€[0,T\, 



(4.20) 
(4.21) 



where, for t £ [0, T], 



qp(*) 



k=l 



if t G [0, u) 



Pi 



if t £ [v,2v), 



0U-FP(t-v),Y,(Pk-Fj;(t-v)) + ), if 



\pr, 



\t-v 



< M + 2, 



fc=i 



*e b'f,0' + J =2,3 



, t»j . . . , 



if 



\\P n \\* t -v > M + 2, 

[j'«,(j + l»,i = 2,3,..., 



(4.22) 
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and 



F i («) = 7= . «e b")U + l)")j = l-2,.... (4.23) 



Let us argue that these equations uniquely define a policy. To this end, consider equations 
P~T8j) . (pL~T9j) . P~2Tj) . (ET221) . P~2"3"j) . along with the obvious relations between scaled and un- 
sealed processes, as a set of equations for X n , D n , T n , P n , B n , C n , F n (and the scaled versions 
X n , D n ), driven by the data (A n , S n ) (equivalently, (A n , S n )), and satisfying the initial condi- 
tion X n (0). Arguing by induction on the jump times of the processes A n and S n , and using the 
causality of the map £, it is easy to see that this set of equations has a unique solution. More- 
over, this solution is consistent with the model equations (|2.ip - (|2.3p . The processes alluded to 
above are therefore well-defined. 

We now show that B n E QS n . To see that B n has RCLL sample paths, note first that, 
by construction, F n , X n are piecewise constant with finitely many jumps, locally, hence so is 
B n . Therefore the existence of left limits follows. Right continuity follows from the fact that 
X n , F n and consequently C n have this property. The other elements in the definition of an 
admissible control hold by construction. Thus B n S 5S n for n G N. As a result, 

V n {X n (0)) < J n {X n {0),B n ). (4.24) 

Our convention in this proof will be that ci,cfc, . . . denote positive constants that do not 
depend on n, e, v, A. Also, the notation (I3.3p - (l3.10p will be used extensively. 
Let, for k = 1, . . . , N, 

i<p k ,e,c k ,™ k ) = {<p[^uM>%<w k ]Mii> k ]). 

Write ip k as (ip k ' 1 ,ip k ' 2 ). Note that (p k is the dynamics corresponding to tp k and C k . Let 
A n = \\A n \\* T + ||S n ||^ and define 

fl£ = {(A n ,S n ) G A k }, k = l,...,N. (4.25) 

We prove the result in number of steps. In Steps 1-4 we shall show that for a constant ci, for 
all n > no(e, v), 

\\X n \\* T < Cl (l + A n ), (4.26) 

and 

sup \\X n - /|| < ae, on f2%, k = l,2,...,N. (4.27) 

M 

The final step will then use these estimates to conclude the result. 

Step 1: The goal of this step is to show (|4.33p below which is the key estimate in proving 
(EOB]) . By Proposition E^ii), 

||C[PlL*<c 2 (l + ||Pi:). (4.28) 

Therefore 

npnn* < J^Sln + ||P m ||*). (4.29) 
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Since pi G (0, 1) for all i G X, we note from (|4.29p that for all sufficiently large n, for any 
||P"|| t %<M + 2 implies - - t,))+ = - i?(t - v)) < 1, 



as Y.i F ii u ) > for a11 u G b> r ]- Define 

f„ = M{t > : \\P n (t)\\ >M + 2}. 

It is easy to check by definition of C™, and using the fact pi G (0, 1) and the convergence 
b n /Vn 0, that for all large n, on the event {f n < v}, 



sup 

tS[0,T] °n 



Pit - / 



< C4. 



Next consider the event {f n > v}. Using ()4.20p . (|4.22p and (|4,29p . one has for all sufficiently 
large n, 

Pi - n r - — , if t G [0,v), 



Hiy/n v 

Pi, if t G [v,2v), 

fH-F?(t-v), if t G [2i;,f„ + t;) 

Pi, if * G [f n + «,T]. 



(4.30) 



Thus, on {f n > u}, 



sup 

te[o,2v] 



Pit 



Jo 



K 



while 



sup 

t£[2v,T] 



Pit - / G?(s)ds 



<c 5 —=+ sup 

V n t6[2u,f»-H>] 



FP(s-v)ds 



2v 



(4.31) 



Consider j > 2 and jv < t < (j + 1)^. Then by the definition of i 7 ^ 
/ F l n (s-v)ds = / F"(s-i;)ds + / F?{s-v)ds 

J2v J2v J jv 



fc[P»]((7 - 2)«) - Ci[F 
6 n t - 



+ 



[ Ci [P"]((j-l»-CjP n ]((i-2»]. (4.32) 
Combining this identity with (|4.28p shows that the last term on (|4.3ip is bounded by 

sup _^4c 2 (l + \\P n \\* t _ v ) < -^=4ca(l + A n ), 

where in the last inequality we also used the fact that T™(t) < t, by which \Df\t = \S?oTp\t < 
\S?\t- We conclude that, for all sufficiently large n, 



sup 

iS[0,T] °n 



Pit- f C?{s)ds <c 6 (l+yT). 
JO 



(4.33) 
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Step 2: We prove (|4.26p . The argument is based on the Skorohod problem (see e.g. [8]) and 
the estimate (fl~33l) . To this end, rewrite $ZjS) as = y™ + Z™, where 

y^(t) = Xf(0)+yft + I?(t)-5r(i;"(t)) + ^^(^- fc?{s)da 

n °n K JO 



n b n j 

Since for each i, X™ is nonnegative and is nonnegative, nondecreasing, and increases only 
when Xf is equal to zero, it follows that (-XJ 1 , ZJ 1 ) is the solution to the Skorohod problem for 
data Y- 1 (see [8] and [7] for this well-known characterization of the Skorohod map (|3.ip ). As 
a result, for all large n, 

\Z?\* T + \X?\* T < 4\Y t n \* T < c 7 (l + A n ), (4.34) 
where we used (|4.33j) and the convergence of /x"/n, Xf (0) and y™. This shows (|4.26[) . 

Step 3: Here we analyze the events f2u, showing that on these events one has, for large n, 
that fii^(pit — Jq C^(s)ds) is close to First, using 

Pl t - T?(t) = Pi t - J* C?(s)ds + |* Cr( S )x { c r(s ) = o}^, 
we obtain from (|4.33p and (|4.34|) . for all large n, 



sup ^^\p it -T?(t)\<c 8 (l + A n ). 

tS[0,T] n °n 

Therefore we obtain that, for all large n, on the event Ufci?^ - , 

sup \ Pit -T?{t)\< V -. (4.35) 
te[o,T] * 

Abusing the notation and writing ^ 2 (T n (-)) for (^i' 2 (T 1 n (-)) ) . . . , ^' 2 (Tf (■))), using (PS"]) 
and (|4.35|) for the choice of u, we have 

sup ||^ 2 (T"(t))-p[^ 2 ](t-^)||< [^(osc 2 „(^ 2 )) 2 j 1/2 <^, (4.36) 
te[v,T] L ^ v / J 4 

on flu, for all n large. 

Next, we estimate S n (T n (t)) - p[ip k ' 2 ](t - v) on J?£. Using pi5| . for all large n, 

sup ||5»(r»(t))-p[/- 2 ](t-t;)|| 

te[u,T] 

< \\S n (T n (-))-^ 2 (T n (-))\\* + sup ||^' 2 (T n (t))-p[^' 2 ](i-7;)|| 

te[u,T] 

4 + 7 4' < 4 ' 37 > 
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where for the first estimate we have used (|4.16p and for second we have used (|4,36D . 

Finally, we show the two estimates (I4.38D and (14.401) below. Note that on fl k one has 
f n > T for all large n (as follows by ||P n ||^ = ||I n |||, + \\D n \\* T < \\A n \\ + < M + 2by the 
discussion in the beginning of the proof (I4.16P ). As a result, (|4.30p is applicable. In particular, 
for all large n, 

IH^ipit ~ [ C?{s)ds) --U = 0, te [0,v). (4.38) 

"n JO ' V 

Now for k = 1, 2, . . . , N, consider 

W£ k (t) := IH^(fHt ~ fc?{s)ds) - CHt -v), te [v,T], 

on the event J?P . We note from (13, 6h that ( k (0) = i. Hence for t G [v, 2v) and all large n, we 
have from P~T5|) and (f3720|) that 

\Wi: k (t)\ = \£ l -CHt-v)\<e. 

Next consider t 6 [2v,T] and integer j for which jv < t < (j + l)v. From the calculation 
(|4.32p . for large n, 

IM^ifHt- f Cr(s)ds) = ii + lMi^ f F?(s-v)ds 

On v JO J °n J2v 

= t l {P n m-2)v) 
+^[c[H((j - w - c,n((j - 2)«)j. 

Hence 

iw&wi < \Ci[P n m - 2)«) - cf (* - «)i + ictn((j - 1)«) - c,[H((j - 2»|. 

For large n, 

|Ci[P n ]((i -2)«)-<f (t-«)| 

< |CJF"]((j - 2)v) - U^\^ 2 o T n ]{(j - 2)v)\ 

+ KJ^' 1 ,^' 2 o T n ]((j - 2)v) - C t [^\p[^ 2 \m ~ 2)v)\ 
+ \Cf{(j-2)v)-CKt-v)\ 

< 3e, 

where the first quantity is estimated using (|4.16|) and (|3.19p , the second using (I4.35P and (|3.19p , 
and the third using (|4.15j) and (|3.20p . A similar estimate gives, for all large n, 

\C i [P n ]((j-l)v)-C i [P n ](V-2)v)\<3e. 

Hence for all large n, on fiV;, 

sup \Wn { t )\ < 6e, (4.39) 
te[v,T] 
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Using flP9l) and (ET33D . for all large n, on 



sup 

t£[v,T] 



n On 







C?{s)ds)-C«(t-v) 



< 7e. 



(4.40) 



Step 4: Now we prove (|4.27p . Recall ip k = ip[ip k ]. The goal of this step is to estimate the 
difference between X n and (p k on Q?. To this end, let first 



<p k (t) 



x + -£ for t G [0, «) 
<p k (t-v) far ie 



Recall from Step 2 that X" solves the Skorohod problem for Y™. Note also that (p k > 0. Thus 
using the Lipschitz property of the Skorohod map we have on Q2 



\Xi -¥i\T< 2 \ Y i ~ It- 
For t G [0, v] and n large, we have, using the definition of Y n and (|4.38p . 

\Yr(t)-$(t)\ 



(4.41) 



< \ X f (0) - xi| + + |If (i) - S?(2f(t))| + 

< c 9 e 



n 



P»i 



Pit - / Cf(s)d S 



(4.42) 



on f2£, where we use (|4.15|) . (|4.16|) and (|4.33|) . Moreover, for t G [v, T], by the definition of Y r ' 

and (p k , 



kS, 



Cf{s)ds 



- Q k (t - v )- Xi - yi (t-v)- ^' L (t -v) + PiW^\{t - v). 

Hence, using P~T5|) . (|4?[6l) . (1437]) and (jOO]) . the estimate (g32D is valid for t G [v,T] as well. 
Namely, |Yj n — (p k \^ < c$e on J?£ for large n. Thus using (|4.4ip . ||X n — </3 fc ||* < c w e on J?£ 
for large n. By the definition of (p k and (|3.6p . (|3.20p . (|4.15p we obtain that, for all sufficiently 
large n, (jOT]) holds. 

Step 5: Finally, in this step, we rely on property (|4.17p to complete the proof. Since (p k is 
bounded, and so is X n on [22, it follows from (I4.27P by continuity of h and g that, for all large 
n, on fl%, 



h{^{s)ds + g^ k {T))-H n <u(s), 



(4.43) 



where 



h(X n (s))ds + g(X(T)), 
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and uj = uja satisfies oj{a) — > as a — > 0, for any A. By (|4.26p and the growth condition on h 
and g, H n < cn(l + A n ). Hence given any A\ > 0, 

H n > Ax implies A n > c^Ax - 1 =: G(Ax). 

Therefore 

E[e b * Hn ] < E[e^ Hn ^]} + E[e b " Hn X{Hn>Al} } 

< E [e b ^ Hn ^} +ne b ^ 1+An h { A->G(A l)} }- (4.44) 

Now we estimate both terms on the r.h.s. of (|4.44p . Denote B = (U^ 1 ^4 fc ) c . Using (|4.43|) . for 
all large n, 

N 

ip-r 6^[fr»A4i]l < Y^Fl P 6 nP"A4l] y . . , 1 _L F [X[^ n A/ii] v . . 1 

i e i - 2-~i t X{(A n ,S")£A k }l + ^\- e X{(A n ,S n )£B}\ 

k=l 

N 

fc=l 

Now by Assumption 12.11 for all large n, 

^logP((I n ,5 n ) G^fe) < - inf I(V) + |, ^logP((I n ,5 n )Gfi)<-infI(^)+e. 
Hence for large n, 

fc=l 

N 

< ^ e ^[/ T / l (^( S ))^+ 9 (^(T))-I(^)+^( e )+ £ ] +e fe2[Z\ 1 -zl+ e ] ) 
fe=l 



where for the first term on the r.h.s. we used (|4.17p and for the second term we used the fact 
B C V C A and the definition of T>a- 

The last term on (|Q4l) is bounded by E[e b " (cil ^ n+Cll+yl "- G( ^ l)) ]. From Assumption E3J 
there exists a constant C12 such that for all large n, 

llogE[e^( c " +1 )^]<c 12 . 

Urn 



Therefore from (I4.44p we obtain 

1 r l2 tjti n 

lim sup -5- log E e n 



< max 

Kk<N L 



T h(<ft(s))ds + <?(/CO) - I(^) + + e 



V [Z\i - Z\ + e] V [en + C12 - G(A 1 )] 
< sup[c(V>, CM) + + e] V [Z\i - Z\ + e] V [cu + c u - G(Z\i)]. 
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Now let e — > first, then A — > oo, recalling that en, cu and G do not depend on A. Finally 
let A\ — > oo, so G{A\) — > oo, to obtain 

limsu P y n (X n (0)) < limsup-^logE[e 6 ^"] < sup c(V>, CM) = V(x), 

where for the first inequality we used (I4.24D and for the equality we used (13. lip . This completes 
the proof. □ 

Remark 4.1 We return to Remark \3.1\f b) regarding a relaxed version of Assumption \2.2l 
where continuous minimizers fh and f g exist. Under the relaxed assumption the proof of 
the lower bound is very similar to the one we have presented. As far as the upper bound is 
concerned, one can define a policy as in the proof of Theorem \4-2[ but with a jump close to 
the end of the interval, to account for the fact that in the solution of the game, the policy 
has a jump at T from a point determined by the minimizer fh to one determined by f g . The 
continuity of the paths ip k is used in the proof of Theorem \4-2\ and so the modified proof will 
have to address the jump at the end of the time interval. This can be done in a manner similar 
to the way we treat the jump at time zero. However, we do not work out the details here. 



5 The linear case 

Section T4. 21 describes a policy for the queueing control problem, that is asymptotically optimal. 
While the construction of this policy and its analysis facilitate the proof of the main result, 
they fail to provide a simple, closed-form asymptotically optimal policy. In this section we 
focus on cost with either h linear and g = or g linear and h = 0, aiming at a simple control 
policy. More precisely, the assumption on the functions h and g is slightly weaker, namely that 

h{x) = s ^c i x i , g(x) = ^diXi, (5.1) 

i=l i=l 

where Cj and d{ are nonnegative constants, and, in addition, 

cV > cV > > c 7 // and dV > > • • • > dtp 1 . (5.2) 

We consider the so-called c/i rule, namely the policy that prioritizes according to the order- 
ing of the class labels, with highest priority to class 1. Let us construct this policy rigorously 
by considering the set of equations 

Biit) = X{Xf(t)>o} ) - B 2'(*) = X{x?(t)=o,x%(t)>o}, ■ ■ ■ ,Bf(t) = X{jq*(t)=o,...,x?_ 1 (t)=o,xp(t)>o}- 

(5.3) 

Arguing as in Section I3~2"l considering (|5.3I) along with the model equations (|2.ip - (|2.3p . it is 
easy to see that there exists a unique solution, this solution is used to define the processes 
X n ,D n ,T n ,B n , and moreover B n is an admissible policy. 

The result below states that the policy is asymptotically optimal. 

Theorem 5.1 Let Assumptions \2.1\ and \2.3\ hold and assume g and h satisfy (|5.ip - (|5.2p . 

Then, under the priority policy {B n } of (15. 3p . 

lim J n (X n (0),B n ) = V(x). 

n— >oo 
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Proof: As explained in Example l2.ll Assumption [272] holds. As a result, the lower bound stated 
in Theorem 14. II is valid. It therefore suffices to prove that limsup n _ s>00 J n (X n (0), B n ) < V(x). 
The general strategy of the proof of Theorem 14.21 is repeated here; the details of proving the 
main estimates are, of course, different. 

Thus, given constants A and e we consider T>a of (|4.14p . M, the constants S±,5, r],Vo,V2, 
the members ij) k of T>a, the sets A k = A v k(ip k ) and the events i?£ (see (I4.25|) ) precisely as in 
the proof of Theorem 14.21 We also set (ip k , £ k , ( k , w k ) = ((p[ip k ], £ [ip k ], C[4> k ], w[^ fc ]) as in that 
proof. 

In what follows, ci, C2, . . . denote constants independent of A, e, 5±, 5, r], vq,V2 and n. 
Analogously to (j4.26j) and (14.27p . we aim at proving that there exists a constant c\, such that 
for all sufficiently large n, 

\\X n \\* T < Cl (l + A n ), (5.4) 
(where, as before, A 11 = \\A n \\* T + ||5 n ||y), and 

sup ||X n -/|| < as, on k = 1,2,... ,N. (5.5) 

[«2,T] 

Once these estimates are established, the proof can be completed exactly as in Step 5 of the 
proof of Theorem 14.21 We therefore turn to proving fl5.4f) and (I5.5D . 

Recall that 6 n = (-£, . . . , -&). Moreover, by Q, V R n = holds if and only if for 



all i, XP = 0, equivalents 9 n ■ X n = 0. Therefore by ([475]) . 

r • z»(t) = ^ (t - jT £ aft-)*) = ^ j[* x { ^ (s)=0} ^ 

Hence from (|2.6p . with 

Y^(i) = If (0) + y?t + - (5.6) 

we have 

6»n . *»( t ) =0n. Y n + ^J t X{en ^ {s)=Q} ds. (5.7) 



Since # n • X n is nonnegative and # n • Z n increases only when 9 n ■ X n vanishes, it follows that 
(9 n ■ X n , 9 n ■ Z n ) solve the Skorohod problem for 6 n ■ Y£. As a result, 



\e n ■ x n \* T + \e n ■ z n \* T < 41(9™ • Y£\* T . 

Also, using ()2.6p . the non- negativity of Xf implies 

Zl\t) > -Y^it). 

Since 9 n — > 9, yf — > yi,X n (0) — > x, it follows that there exists a constant c\ such that for all n 
large, (|5.4p holds, as well as 

\\Z n \\* T <ci(l + A n ). (5.8) 
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Toward proving (|5.5p , let us compute the paths tp k . As mentioned in Example 12.11 the 
corresponding minimizing curve is given by f(w) = (0, . . . , 0, wfi 1 ), w > 0. Recall the notation 
([OD and that £ k = $[if; k }. Thus 

fo, if i = 1,2, ... , J- 1, 

^ = (5-9) 
l/i J r[0-£*], if * = J. 



Define X' = {1, 2, 1} and p' = X^ti Pi- Then by $ZM) and (127FL 



ieX' iei> n iez 1 

= U n {t) + ^ f X { xn,, is)=0} ds, 



n JO 



where 



and we used (|5,3p by which Y2i> = holds if and only if Xf = for all i £l'. Hence, 
invoking again the Skorohod problem, 

X n >'(t) = U n (t) + sup {-U n {s) V 0}. (5.10) 
se[o,t] 

We will argue that, on ,J? n := U^l?]? , for all sufficiently large n, 

sup \X' n \ < c 2 e. (5.11) 

To this end, let us fisrt show that, for all sufficiently large n, the following holds: On i? n , 
U n (t2) < U n (t\) whenever £i,i 2 G [0, T] are such that £2 — *i > i>2- Suppose this claim is 
false. Then there are infinitely many n for which there exist (n-dependent) £i,t 2 £ [0>^1 with 
£ 2 - ii > u 2 but C/ n (i 2 ) > U^ih) on I? n . Thus 



E o?[xm + + A?(ti) - stcto))] 



E fl?[A?(0) + tfta + i?(ta) - ST(2T(fe))] 



<^(p / -l)fe-£i)<^(p / -l)^ 2 . 

However, this is a contradiction because the r.h.s. tends to —00 as n — > 00 whereas the l.h.s. 
remains bounded. This proves the claim. 

Next, note that, for a similar reason, for all sufficiently large n, U n {t) < on Sl n , for t > t> 2 . 
Hence for t > u 2 and n large, we have on 17™, 

sup {-f/ n (s) V 0} = sup {-U n (s)} = sup {-[/"(s)}. 

se[o,t] se[o,t] se[t-«2i*] 
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Thus using (|5.10p . on we have for all n large and t > V2, 

X n >'(t) = U n {t)+ sup {-U n (s)} 

se[t-v 2 ,t] 



^^ i (t) + ^-l)t+ sup \-^Yl t (s)-^{p' -l 

;<=.ti ° n [t-v 2 ,t] , cT , °n 



— 

< Y, e i Y U t ) + su p f-E^^wl 

< c 3 e + c 3 [osc j;2 (i n ) + osc j;2 (S n )], (5.12) 



where we used (|5.6p and the fact that T™ are Lipschitz with constant 1. On HP, 

osc„ 2 (I n ) < 2\\A n - ifj k ' l \\* +osc„ 2 (^ fc ' 1 ) < 3e, (5.13) 

where we used (|4.16p and (I4.15p . Similarly, osc„ 2 (5' n ) < 3e. Using this in (15.121) gives (15. lip . 

Next, recall that 9 n ■ X n = r[9 n • Yg]. Note by J5I2J that 9 ■ (p k = r[0 • £ k }. Therefore using 
the Lipschitz property of r we have, for all sufficiently large n, 

\9 n ■ X n - 9 ■ tp k \* T < 2\9 n -Y£-9 n ■£% + 2\\6 n - 0\\\\£ k \\* T 
<c4Y£-t k \\* T + e 

< c 4 J2i\ A i ~ + 1^ Tt - P^AYt} + 2e. (5.14) 

i 

Now, on J2£, || A n - ip k ^\\ < e and \\S n - ip k ' 2 \\ < e. Moreover, from (|5.8p . 

svop\{ Pl t-T?{t))\ <v 2 , 

[0,T] 

on fi n . It follows that, on f22, for all sufficiently large n, 

\6 n ■ X n - 9 ■ (p% < c 5 e + osc^^' 2 ) < c 6 e, (5.15) 

where the last inequality follows from (|4.15p . 

Now, by (|5.1ip and the fact that <p k = for % < I (see (15. 9p ). we have sup[„ 2i T] \X™ — <p k \< 
c-jE for i < I, on i?^ - for large n. Combining this with (I5.15p . the convergence 9 n —> 9 and the 
fact that the I vectors 9 and {ei,i < 1} are linearly independent, gives sup[„ 2iT ] ||X n — ip k \\ < 
cs£, on [22, for all sufficiently large n. This proves (|5.5p and completes the proof of the result. 
□ 



A Appendix 

Proof of Proposition I2.lt We borrow some ideas from the proof of Lemma A.l in |22j . 
Clearly, the statements regarding A n and S n are identical, hence it suffices to consider only 
the former. Define M\{u) = E[e uL4i ] for mGR, It suffices to prove that for any positive K > 
and i € Z, 

limsup-^logEle^ 1 ^ 1 *)] < oo. 
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Assume i = 1. Since M\{u) = ~K[e uIA ] is finite around 0, it is C 2 there, and so is H\(u) := 
log M\(u). Therefore by Taylor expansion there exist 7, 5 > such that 

\Ha( u ) ~ u \ — 7 n2 > f° r an u with \u\ < 5. (A.l) 

Here we have used the fact that ^f(O) = E[L4*] = 1. Note that 

E[e ^|A5T)] 

= 1 + b 2 n K J e 6 « Kt P(|i?|* > t)dt < 1 + b 2 n Ke KK + b 2 n K J <*« Kt P(\A?\* > t)dt. 
For t > 1, 

P(|A>|* > t) = F(3v G [0,T] such that > t) 

< P(3t> G [0, T] such that < -*) + P(3u G [0, T] such that > 

Now 

< -t < -b n ^t + A>. 

Let L^J denote the largest integer less than or equal to x. Also assume —b n \fnt + A™T > 0. 
Then 

F(3v G [0,T] such that AJ(t;) < -t) 
= F(3v G [0, T] such that A?(v) < -b n <Jnt + A» 

< F(3v G [0,T] such that ^ /A 1 ^) > A» 

1=1 

[-bn^/nt+Xfv+lj 

< F(3v G [0, T] such that ^ (^(O - 1) > A?w - + A?« + lj ) 

L-6„Vnt+A?v+lJ 

< P(3-u G [0, T] such that ^ (IA l {l) - 1) > b n ^t - I). 

1=1 

We define 14 = ^f =1 (ZA 1 (Z) — 1). Then {V/J is a martingale w.r.t. the filtration generated by 
{IA l (l)}. For all large n and t > 1, & nA /n* - 1 > 0. Denote L n = |_-&n\/ni + A"T + lj . Then 

F(3v G [0, T] such that A?(y) < -t) < P( sup \V k \ > b n ^t - 1) 

\<k<L n 

< e -^(6„^-l) E [ sup e /3n|^l] 5 
l<fe<L" 

where /3 n > are any constants. We note that {e^'^'jfc is a sub-martingale. Hence by Doob's 
martingale inequality 

E[ sup e^l^l] < E[ sup e 2A, l Vh l]s < 2E[e 2ftl|yin| ]i 

Kk<L n Kk<L n 
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Thus 

F{3v e [0,T] such that A^(v) < -t) 

< 2e _/3?l< ' bnv/ "* _1 ^E[e 2/3 ' l ' v ' i ™']5 

< 2e _/3n ^™ v/ ™* _1 ' ) [E[e 2/3nV ' L ™] + Efe -2 ' 3 "^"]]^ 

< 2e -Pn(b n Viit-l)r e L n (Hl(2/3 n )-2l3 n ) + e L n {H\ (-2/9 n )+2/9 n )i | _ 

If 2/3 n < 5 and n is large enough so that bn ^ lt — 1 > holds then using (jA.ip we have 
P(3w € [0,T] such that A?(v) < -t) < 2y/2e~ Pn ■ £ ^ e 4L " 7 ^ 

Now we choose /3 n = fy(2K + 2) and we choose n\ such that for n>n\, 2fi n < 5. Then 

¥(3v G [0,T] such that A?(u) < -t) < 2v^e£ 16 *^^* +1 )V#(* +1 > t . (A.2) 
In a similar way we obtain ri2 such that for all n > ri2 

¥(3v e [0,T] such that > t) < 2\/2V' 16 ^ 7 ^ +2 )V 6 »^ +1 )*. (A.3) 

Thus from (jA.2j) and (jA.3|) we have constants ^3,71,72 such that for all n > 713, P(|A"|* > 
t) < -y 1 e fe " 72 e _f) ™^ +1 ^. Hence for n > 713, 

/"OO /"OO 1 

/ e^'PriA?!* > t)dt < 7ie 6 ™ 72 / e" 6 ™^ = -^71 e 6 ^ 72 " 1 ), 

Ji Ji K 

and K[e b ™( K \ A ™ 'I*)] < 1 + b^Ke Kb " + ^71 e 6 ™^ 72-1 ), which gives the required estimate. □ 

Acknowledgement. We are grateful to the referees for constructive comments and sugges- 
tions that have much improved the exposition. 
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